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ABSTRACT 

The study compared the effects of a professional development course delivered in an online and a face-to- 
face format. The effects examined included changes in teachers’ pedagogical beliefs, instructional 
practices, and understanding of teaching numher-sense and related mathematical concepts. The study 
randomly assigned participants to either the online or the face-to-face format and employed the same 
instructors, reading material, and instructional activities for both formats of the course. Both formats of 
the course were also delivered over the same eight-week period and required participants to invest 
approximately the same amount of time each week engaging in learning activities. Both formats of the 
course showed significant impacts on teachers’ mathematical understanding, pedagogical beliefs, and 
instructional practices. Consistent with prior research on online versus face-to-face instruction, the 
positive outcomes were comparable across both formats. Interestingly, teachers who participated in the 
online course reported that they were more willing to take courses in the future online than did teachers in 
the face-to-face condition. Further research is needed to determine whether this finding is limited to self- 
selected teachers, the specifics of this course, or other factors that limit generalizability. 
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I. INTRODUCTION 

Online professional development provides several advantages over face-to-face sessions. In an online 
course, teachers have more flexibility and greater choice as to when they participate and engage in learning. 
Teachers in remote areas or who work in small schools can also gain access to professional development 
courses that would otherwise be expensive or impractical to deliver in a face-to-face environment. Through 
online courses, it is also easier to connect teachers across schools and districts, thus widening perspectives 
and fostering professional connections that would not occur otherwise. Nonetheless, since its introduction 
more than a decade ago, educational leaders have questioned whether professional development delivered in 
an online environment is as effective as traditional face-to-face sessions in increasing teachers’ content and 
pedagogical knowledge and in improving their instructional practices. 


The study presented here builds on past research that examines the effect that different design features of 
online professional development has on teachers’ knowledge, pedagogical beliefs, and instmctional 
practices [1, 2]. These previous studies systematically varied the level of interactions that occurred within 
different versions of a professional development course. One version included asynchronous, facilitated 
discussions among participants as a central, required part of the course while in the other version each 
participant worked independently, without a means of communicating with their peer participants. These 
studies also manipulated the amount and types of interactions participants had with the course instructors. 
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At one extreme, participants had access to two instructors, one with content area expertise and the other 
skilled in facilitating online discussions. Participants in this strongly facilitated online course received 
considerable amounts of feedback and had interactive discussions with the instructors throughout the course. 
At the other extreme, participants worked through the course at their own pace with minimal interaction 
with a single facilitator who handled technical and logistical questions but did not have content expertise. In 
this version of the course, participants were instructed to complete assigned readings, classroom activities, 
and writing assignments on their own and received a simple statement that their writing assignment was 
received and met the course requirements, with no constructive feedback or follow-up questions. 


In both of these previous studies, the purpose of manipulating the level of interaction among participants 
and the amount and type of interactions with the course instructor(s) was to examine whether these design 
features influenced the intended outcomes of the course. All versions of the course were found to have 
comparable positive effects on teachers’ pedagogical beliefs, instructional practices, and knowledge of 
teaching algebra (the focus of the course). Surprisingly the positive effects did not differ significantly among 
the different versions of the course. 


As described more fully below, the study presented here shifts the focus of our research from manipulating 
aspects of an online course to directly comparing the effects of online versus face-to-face courses in which 
the goals, content, and activities are kept as comparable as possible. In the sections that follow, we provide a 
brief review of prior research on online professional development that informed the design and focus of the 
study presented here. We then describe the course and the design of the face-to-face and online versions of 
the course. We then describe the methodology employed to recruit and assign teachers to groups, as well as 
the data collection instruments employed for this study. Finally, we present the study findings and explore 
the implications these findings may have for the design and delivery of online professional development. 


II. SUMMARY OF PAST RESEARCH 

Since the early 1980’s, there has been a lack of consensus about whether online instruction affects learning 
as well as face-to-face instruction. Believing that technology is a neutral carrier of content, Clark [3, 4] 
argues that the use of a given technology for instructional purposes should not affect learning outcomes. 
Other researchers, however, argue that computer-based technologies have potential to be transformative [6, 
7, 8]. In the context of professional development for K-I2 teachers, the use of technology to delivery 
professional development courses in an online forum offers several potential advantages over face-to-face 
instruction. 


In contrast to the spontaneous verbal communication of face-to-face learning environments, asynchronous 
learning environments in which communication is primarily written may provide conditions that promote 
reflective inquiry [9, 10, 11]. Threaded discussions available in online environments differ from face-to-face 
discussions in that they enable exchanges across time and space, provide a permanent record of interactions, 
and allow participants more time to reflect on a given topic before responding [9]. While there are elements 
of real-world conversations in threaded discussions, there is nothing analogous to threaded discussions in 
the real world [12]. 


In a 5-year longitudinal case study, Delfino and Persico experimented with different techniques for teacher 
professional development including a face-to-face approach, an online approach, and a blended approach 
[13]. Although the general objectives of the face-to-face and online courses were the same, Delfino and 
Persico found the online approach, which was free from time constraints and maintained a record of written 
interactions, promoted critical thinking and in-depth reflection on content. Similarly, when comparing 
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discussions of graduate students in an online environment with those in a face-to-face environment, Meyer 
found that the use of threaded discussions increased the amount of time learners were engaged in 
discussions related to class ohjectives [14]. Meyer indicates that learners in an online medium spent more 
time reflecting on course material, asking questions, and engaging in discussions about the course content. 
Given the speed with which face-to-face discussions occur and evolve, Meyer also suggests that online 
threaded discussions may benefit learners who require additional time and reflection to construct 
knowledge. 


Despite the potential benefits that online discussions may have for supporting student learning. Garrison 
argues that when online learning simply recreates the conditions of a traditional classroom, it fails to exploit 
the unique strengths of online learning [9]. While researchers have reached a broad consensus on the general 
components of high-quality K-12 teacher professional development in face-to-face contexts, many 
questions remain about the design and delivery of effective online professional development [15]. 


Several recent studies have focused on the characteristics of existing OPD programs, but little research has 
examined specific OPD formats and their impacts on teacher practice and student learning [16, 17, 18]. 
Ginsburg, Gray, and Levin reviewed the quality and effectiveness of more than 40 online professional 
development programs, judging them against what is known about similar evidence for effectiveness and 
quality in traditional face-to-face professional development [17]. Overall, Ginsburg et al. found that the 
OPD sites lacked rigorous independent assessments of the impact of OPD on teacher or student outcomes. 
Similarly, in a review of the literature on professional development for teachers. Lawless and Pellegrino 
concluded that success of professional development is typically judged from anecdotal evidence or the 
results of surveys in which teachers indicate their satisfaction with the experience or their perception of its 
usefulness to their work [16]. Consequently, researchers know what teachers think about their professional 
development activities, not what they actually learn, how it changes their pedagogies, or how it impacts 
student learning [16]. 


Two recent studies, however, systematically varied specific characteristics of a professional development 
course and examined the impact on learning outcomes. In both studies, the learning goals, the course 
materials, and the assignments participants were required to complete were identical. In the first study, the 
course was delivered in two formats. For the first format, the course was led by a course facilitator who 
actively participated in discussions, guided participants through the course material in a paced manner, and 
encouraged interactions among participants through a threaded asynchronous discussion forum. For the 
second format, participants did not have access to a threaded discussion board, were unable to communicate 
with other participants via email, had limited discussions with the course instructor, and completed the 
course material at their own pace [1]. 


The second study contrasted four conditions. One was a highly supported condition, with a math education 
instructor, an online facilitator, and asynchronous peer interactions among participants as they worked 
through the course together. Another was a self-paced condition, in which participants did not have access 
to a math education instructor, could not communicate with other participants through online discussions or 
email, and simply submitted assignments to the instructor who did not provide any specific feedback to 
participants. A third condition included threaded discussions but were led by an online facilitator who did 
not have expertise in math education instruction. The fourth condition allowed participants to communicate 
online with an instructor with expertise in math education instmction, but participants were not able to 
communicate with each other [2]. Both studies found that all versions of the course had positive effects on 
teachers’ beliefs about math instruction, on their practices (as reported by themselves and their students), 
and on their understanding of mathematics instruction as measured by a mathematics test. Surprisingly, both 
studies also found that the positive effects did not differ among the conditions. Regardless of the level of 
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facilitation, the level of expertise of the instructor, the pacing of the course, or the level of interaction among 
participants, the course had the same positive effects on teachers’ pedagogical beliefs, instructional 
practices, and increased knowledge of teaching mathematics [1,2]. 


This finding parallels findings from several studies that have compared the effects of traditional face-to-face 
instruction versus online instruction. Although small, this body of research consistently reports that no 
significant difference in learning outcomes occur between courses delivered in face-to-face or online modes 
[19, 17, 20, 21]. The study presented here extends the two previous studies that compared the effects of 
various online conditions, and seeks to expand the body of research that compares the effect that the mode 
of delivery, that is online versus face-to-face, has on the intended learning outcomes of the course. 


III. BUILDING A SYSTEM OF TENS COURSE 

The course employed for this study was titled Building a System of Tens. The course was originally 
developed by Deborah Shifter and her colleagues and is Part I in a series called Developing Mathematical 
Ideas (DMI). For the purposes of this study, the face-to-face version of this course was adapted by the 
Education Development Center to create an online version. 


For the study presented here, both the online and face-to-face versions of the course required teachers to 
read the same material, experiment with the same activities in their classroom, and complete the same 
assignments within an eight-week period. In addition, to control for the potential effect of instructor quality, 
the courses were run by the same set of facilitators, each of who facilitated one face-to-face session and one 
online session. The intended learning outcomes of the course were also the same for both versions. These 
outcomes included the following: 

Increased understanding of number sense. 

Increased ability to identify misconceptions and errors that students apply when working on problems 
specific to number sense. 

Increased ability to identify and apply instructional strategies that hold potential to help improve student 
understanding of these concepts. 

Increased use of instructional methods that allow students to develop their thinking, share their 
understandings, and discuss their classmates’ problem-solving strategies. 


What differed between the two versions of the course was the mode in which the course was delivered. In 
the version we term “online”, participants logged on to the course website to access course materials. The 
course website’s discussion board provided the platform for participants to communicate with other course 
participants and the course facilitator. The online course followed a weekly schedule and participants were 
expected to spend three hours each week interacting online with their peers and completing course activities. 
Each online cohort had two course facilitators: one content expert and one online facilitation expert. The 
content facilitator provided individualized, instructional feedback on assignments and other direct 
instruction as needed. The content facilitators where experienced mathematics educators who had been 
trained in the DMI approach by the original course authors. The primary duties of the online facilitation 
expert were to send welcome and twice-weekly reminder e-mails, track participation, and provide technical 
assistance when the need arose. Course participants were given the email addresses and phone numbers of 
their course facilitators. The content facilitator followed the same syllabus employed by the face-to-face 
participants. Participants completed assignments weekly and submitted them to their content facilitator. 
These participants downloaded all course material and viewed videos online. 
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In the second version, which we term “face-to-face”, participants attended weekly class sessions throughout 
the 8-week duration of the course. Each class lasted for approximately three hours. The face-to-face cohort 
had one course facilitator trained and experienced in DMI. Participants were given the email address and 
phone numbers of their course facilitators. While some participants contacted their facilitator via phone and 
email, the vast majority of the interactions occurred during class periods. The course facilitator followed a 
syllabus that was identical to that used for the online version. The facilitator provided individualized, 
instructional feedback on assignments and other direct instruction as needed. Participants were expected to 
complete weekly assignments and bring these assignments to class. Participants spent much of the class 
sessions working in small groups with other teacher participants. Some teachers independently exchanged 
email addresses with one another and communicated outside of class. 


It should be noted that participants in both versions of the course were awarded graduate course credit and a 
stipend of $200 for participating in the course and completing all of the data collection instruments 
described below. 


IV. STUDY DESIGN 

The study presented here was designed to compare the learning outcomes of the same course delivered 
online or face-to-face. To limit class size, a total of six sections of the course were delivered, three online 
and three face-to-face. In addition, three facilitators with content-area expertise were hired to run the 
sessions. To control for the effect that a facilitator may have on the learning outcomes of the course, each 
facilitator ran one online version and one face-to-face version. 


Participation in the study was limited to teachers in grades one through five who were currently teaching 
mathematics. Because one version of the course required teachers to meet face-to-face, three geographic 
areas were targeted, namely schools within easy driving distance of Bourne, MA, Newton, MA, or 
Manchester, NH. Principals of schools within easy driving distance of the targeted geographic areas were 
asked to invite teachers in grades 1 through 5 to participate in the professional development course. One 
hundred and fifty teachers who met the following criteria were selected to participate in the study: 
sCurrently teach math to elementary level students, and; 

Have a working email address, and 

Have access to a classroom of students with whom to work. 


Teachers were stratified by geographic location and gender, and then randomly assigned to one of the two 
treatment groups: face-to-face or online. The face-to-face cohorts contained 21 teachers in Manchester, 16 
teachers in Bourne, and 18 teachers in Newton. The online cohorts contained 28 teachers in Manchester, 35 
teachers in Bourne, and 32 teachers in Newton. Note that the online versions were assigned a larger number 
of teachers because prior research indicated that approximately 40% of teachers taking the online courses 
did not complete the course. While we did not have data on the drop-out rate for face-to-face courses, we 
assumed that the drop-out rate would be smaller for face-to-face. Thus, to yield approximately the same 
number of participants in both settings at the end of the study, a larger percentage of teachers were randomly 
assigned to the online courses. 

V. DATA COLLECTION INSTRUMENTS 

The study presented here focused on the effect that the two versions of a professional development course 
had on the intended outcomes of the course. To collect data about the intended outcomes, six instruments 
were employed. These instruments included a background survey, a pedagogical beliefs and practices 
survey, a measure of teachers understanding of teaching the base ten number system, a student survey, a 
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teacher log, and a course evaluation. As shown in Table 1, the background survey, pedagogy survey, 
teacher log, and the mathematics assessment were administered prior to the start of the course. The 
pedagogy survey, mathematics assessment student survey, teacher logs, and course evaluation were 
administered after the course was completed. Below each of the instruments is described briefly and pdf 
versions of the instruments are available at http://www.bc.edu/research/intasc/researchproiects 
/optimizingOPD/OPD.shtml . 


Table 1: Instrument List 



Administered 

Administered 

Instrument 

Pre-course 

Post-course 

Background Survey 

X 


Pedagogy Survey 

X 

X 

Math Assessment 

X 

X 

Student Survey 

X 

X 

Teacher Log 

X 

X 

Course Evaluation 


X 


A. Background Survey 

The background survey was designed to collect demographic information and information about each 
teacher’s prior experiences with professional development and technology use. The majority of the items on 
the background survey were closed-response. In a few instances, participants were requested to type in 
numeric values. In addition, the background survey contained items that were used to form four scales. 
These scales and their associated reliabilities included Prefer to Work on a Team (.78), Prefer to Learn on 
Own (.69), Proactive Regarding Own Professional Development (.56), and Access to and Use of 
Technology (.70). Each of these scales was standardized to have a mean of zero and a standard deviation 
equal to one. 


B. The Pedagogy Survey 

The pedagogy survey collected information about teachers’ pedagogical beliefs and instructional practices, 
and was administered prior to and following the completion of the course. All items were closed-ended and 
asked teachers to either report the frequency with which they employed specific instructional techniques or 
to indicate the degree to which they agreed or disagreed with a statement that focused on the value of a 
given instructional practice. The vast majority of items employed for this survey were taken from two 
previous studies that focused on the relationships between pedagogical beliefs and practices and the use of 
instructional technology in the classroom [22, 23]. The specific scales that were formed and the associated 
reliabilities included Teacher Confidence Teaching Mathematics (.84), Student-Centered Beliefs (.80), 
Teacher-Directed Beliefs (.56), and Comfort Recognizing Students Problem Solving Strategies (.86). 
Scores for each scale were formed by calculating each teacher’s mean response using a four-point scale that 
ranged from 1 (low score value) to 4 (high score value). 


C. Math Assessment 

The math assessment was designed to collect information about teachers’ understanding of the base ten 
number system and related concepts covered in the professional development course. The assessment was 
administered twice, once in the first week and once in the final week of the course. The assessment 
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included a set of multiple-choice items and open-response items that presented teachers with a sample of 
student work for a given problem specific to the base ten number system. The teacher was then presented 
with two sets of open-response items. The first set of three open-response items presented examples of 
student work and asked the teacher to explain the method the student used to acquire their answer and to 
evaluate whether the method makes mathematical sense. The second set of items presented the teacher with 
work performed by two students and asked the teacher the following questions: a) Explain what the 
students might have been thinking. Why does this method give the wrong answer? and b) As their teacher, 
what is an approach that you might take to help students understand why this method doesn’t work? Include 
examples to clarify your approach. Using a scoring guide that was shown to yield reliable scores (inter-rater 
reliability of .80 or higher), two readers independently scored teacher responses and, when discrepancies 
occurred, the readers discussed their scores before reaching a consensus score. The scoring guide 
employed the following four-point scale: (0) does not meet expectations; (1) partially meets expectations; 
(2) meets expectations; and (3) exceeds expectations. 


D. Teacher Logs 

The teacher logs were designed to capture information about teachers’ day-to-day pedagogical practices. 
The teacher logs were administered twice, once during the first week of the course and once during final 
week of the course. During each administration, teachers completed three logs. Each log consisted of a 
series of instructional strategies similar to those explored in the course. For each strategy, teachers were 
asked to indicate whether the strategy was: (1) not used at all; (2) a minor component of the lesson; (3) a 
major component of the lesson; or (4) the most important component of the lesson. The ratings provided for 
each activity were then averaged across the logs recorded for each week. 


E. Student Survey 

To triangulate data provided by teachers via the pedagogy survey and the teacher logs, a survey was 
administered to students in the teacher’s mathematics classroom. The study survey was administered twice, 
once prior to the start of the course and once following the completion of the course. The survey asked 
students to indicate the frequency with which they engaged in specific learning activities (e.g., performing 
worksheets individually, working with partners to solve problems, sharing solutions with their class, etc.) 
and how often their teacher employed specific instructional strategies (e.g., asking students to explain how 
they solved a problem, showing students how to solve a problem, asking students to respond to each others 
work, etc.). All items were forced-choice and were used to measure the frequency with which students and 
their teachers engaged in specific instructional and learning activities. The scale employed for these items 
ranged from: (1) almost always; (2) most of the time; (3) once in a while; and (4) never. Hence, a low score 
for a given item indicated more frequent use of the given instructional strategy. 


F. Course Evaluation 

The course evaluation was designed to collect information from teachers about the positive and negative 
aspects of the course. The course evaluation instrument asked participants to rate the value and quality of 
various aspects of the course including the reading material, the assignments, the facilitator, feedback on 
assignments, etc. 

VI. FINDINGS 

The primary research question examined in this study asked whether the mode of delivering the course — 
online versus face-to-face — affected teachers’ mathematical understanding, their pedagogical beliefs, and 
their instructional practices. To this end, the majority of analyses conducted for this study focused on 
comparing the effect of the two versions of the course on these three outcomes. But, to provide a better 
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understanding of characteristics of the study participants, descriptive statistics were calculated for several 
items on the background survey. We begin, then, with results of the background survey. 


G. Characteristics of Study Participants 

The majority of teachers who participated in and completed the study were female (95%). The vast majority 
of teachers also identified themselves as white (93%). The age of participants was distributed fairly evenly 
across age groups, with 6% 25 years or under, 11% 26-30 years, 11% 31-35 years, 13% 36^0 years, 12% 
41-45 years, 19% 46-50 years, 15% 51-55, and 14% older than 55. There were wider differences in the 
number of years participants reported that they had worked as a teacher. Six percent had taught for less than 
one year, 26% for 1-5 years, 18% for 6-10 years, 21% for 11-15 years, 8% for 16-20 years, 9% for 21-25 
years, and 12% for more than 25 years. The vast majority of teachers reported that they did not have either 
a minor or a major in mathematics (93%), but did have full credentials for teaching in an elementary school 
(84%) and had a Masters degree (66%). Sixty-two percent of teachers also reported taking 1-5 professional 
development courses related to mathematics during the previous three years. 


A t-test was conducted on each of these background variables and on the four background scales to examine 
whether the treatment groups differed with respect to their demographics and prior educational and work 
experiences. No statistically significant differences between the groups were found for any of the 
background variables. 


H. Completion Rate 

Of the 150 participants who agreed to participate in the study, 24 opted not to participate once they were 
notified of the treatment to which they were assigned. No data was collected from teachers who opted not 
to participate in the study. However, it is interesting to note that several of the teachers who opted not to 
participated noted that they were hoping to take the course online. 


The actual number of teachers who began the study was 126. Of those, an additional 45 teachers either did 
not complete the course or did not complete the data collection instruments. Thus, of the teachers who 
initially agreed to participate in the study, 54% completed all components of the study. Of those teachers 
who actually began a course, 64% completed. While these completion rates are lower than we anticipated, 
they were consistent with past research on online professional development. As an example, the drop out 
rates for the two studies upon which this research builds were 46 and 44% [1,2]. 


Table 2 displays the drop-rates for each session of the course. For two of the three face-to-face sessions, the 
drop-out rate was noticeably lower than the online courses. However, the third face-to-face cohort 
experienced the largest drop-out rate of all sessions. 
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Table 2: Drop-out Rates within Each Session 



Face-to-Face 

Online 


Manchester 

Bourne 

Newton 

Manchester 

Bourne 

Newton 

Drop-out Rate 

14% 

20% 

67% 

32% 

60% 

47% 


Across cohorts, participants who did not complete the course were sent a survey inquiring about the reasons 
for their non-completion. Twenty-one drop-outs responded. Thirty percent of those drop-outs who 
responded indicated that personal reasons, such as family health issues or divorce, made it too difficult to 
participate in the course. Twenty-four percent of respondents indicated that the demands for the course 
were too high and an additional 19% stated that the time commitment was larger than they had anticipated. 


Chi-square analyses were performed to examine differences in background characteristics for students who 
dropped-out and those who participated. Statistically significant differences were detected for four items. 
First, while 33% of those who completed the course reported that they had taken a course on the base ten 
number system previously, 67% of drop-outs had done so. This may indicate that the content of the course 
was more familiar for those who dropped out than it was for those who persisted. Second, a higher 
percentage of teachers who persisted through the course stated that they tend to complete tasks and 
assignments before deadlines. Third, a higher percentage of teachers who persisted through the course 
stated that they strongly value learning by seeing visual images such as pictures or movies. Finally, teachers 
who dropped-out of the course reported that they had taken more professional development courses related 
to teaching mathematics over the past three years as compared to those teachers who persisted. It is 
important to note that no statistically significant differences were found for characteristics such as 
preference to work on a team, preference to learn on own, being proactive regarding own professional 
development, and access to and use of technology. Finally, of those participants who dropped out, none of 
the background characteristics differed significantly between those who were in the face-to-face or the 
online version of the course. 

I. Quality of the Course 

Overall, participants believed that both versions of the course were well developed, well delivered, and 
valuable. Across all items on the survey that asked participants to rate the value of specific components of 
the course and the course overall using a four point scale that ranged from not valuable to very valuable, the 
mean fell between the valuable and very valuable. As a few examples, when asked to rate the value of the 
case studies, the mean rating was 3.67 with 4 representing “very valuable.” When asked to rate the value of 
the expertise and feedback from the facilitator, the mean rating was 3.45. And when asked to rate their 
learning in the course compared to their previous face-to-face professional development opportunities, the 
mean rating was 3.82. 


To examine whether participants in the online and the face-to-face groups had similar views about the 
course, a series of t-tests were conducted for each item on the end-of-course evaluation. After adjusting for 
multiple comparisons, a statistically significant difference was found for five items. Specifically, as one 
might expect, teachers in the online version of the course reported experiencing more technical problems 
while taking the course than did teachers in the face-to-face group (t=3.20, p<0.01). Teachers in the face-to- 
face version also reported a higher value for performing the assigned math activities (t=4.72, p<0.01), 
sharing ideas with other participants (t=3.18, p<0.01), and receiving feedback from their facilitator {t-3.21, 
p<0.01). For all three of these items, however, the mean rating for both groups was above 3, indicating that 
both groups believed each of these were valuable. The final difference occurred on an item that asked, “If 
given a choice, I would prefer to take a course like this face-to-face as opposed to online.” The scale for this 
item ranged from 1 (strongly disagree) to 4 (strongly agree). The mean rating for teachers in the face-to- 


Joumal of Asynchronous Learning Networks, Volume 13: Issue 2 


79 


Face-to-face and Online Professional Development for Mathematics Teachers: A Comparative Study 


face group was 3.41, while the mean rating for the online course was 2.40 (t=4.41, p<0.01). Clearly, those 
teachers who experienced the course in a face-to-face environment indicated that they would prefer this 
version in the future. In contrast, teachers who experienced the online version seemed less enthusiastic 
about taking a similar course in a face-to-face environment. 


J. Changes in Pedagogical Beliefs 

The Pedagogy Survey was designed to collect information about teachers’ pedagogical beliefs. The survey 
was administered pre-and post-course, and the data was used to examine changes in teachers beliefs and 
practices that followed their participation in the course. It should be noted that the scale scores ranged from 
I (low value) to 4 (high value). Table 2 displays the mean scale scores for each treatment group for the pre- 
and post-course administrations. The column labeled Change displays the change in mean score between 
the pre-course and post-course administrations. For the teacher confidence teaching math, student-centered 
beliefs, and comfort recognizing students’ problem solving strategies scales, teachers scale scores increased 
significantly. While there was a slight decrease in teacher-centered beliefs, this change was not statistically 
significant. There were no statistically significant differences between the face-to-face and online groups 
for any of the pedagogical belief scales. 


Table 2: Scale Score Difference between Pre- and Post-survey, Grouped by Treatments 



Pre 

Post 

Change 
(Post - Pre) 

P 

Teacher confidence teaching math 

2.85 

3.39 

.54 

<.01 

Teacher-directed beiiefs 

2.58 

2.52 

-.06 

0.50 

Student-centered beiiefs 

3.75 

4.15 

.40 

<0.01 

Comfort Recognizing Students’ Probiem 
Soiving Strategies 

2.13 

2.95 

.82 

<0.01 


K. Changes in Instructional Practices 

Teachers completed three teacher logs prior to the course and three logs following the completion of the 
course. As described above, the logs asked teachers to indicate the extent to which each type of activity was 
a component of the logged lesson. Table 3 displays the mean ratings given by teachers across lessons logged 
prior to and following the course. The column labeled post-pre indicates the change in the mean rating 
between the pre-course logs and the post-course logs. Within the face-to-face group, the extent to which 
these twenty-seven instmctional strategies were elements of lessons changed significantly for fifteen 
strategies. Two of these strategies — having students do worksheets and stopping class to address 
discipline — saw declines in their occurrence. The remaining thirteen strategies experienced increases. 
Among the strategies that experienced increases were: developing and adapting instructional material; 
asking students to make conjectures, draw pictures, write about math, work with manipulatives, or solve 
problems related to everyday life; leading whole class discussions and asking follow-up questions to 
students’ responses; and asking students to work in pairs or groups to solve problems. For the online group, 
significant changes were found for sixteen strategies. Half of these strategies were the same as those that 
saw changes in the face-to-face group. Only one strategy experienced a significant decrease — shaving 
students work independently. The remaining strategies experienced increases. 


Independent sample t-tests were conducted to compare the amount of change that occurred for each item 
between the two groups. After adjusting for multiple comparisons, a significant difference was found for 
only one item — shaving students work independently. For this item, the face-to-face group experienced a 
small increase in reported use while the online group reported a significant decrease. The fact that a 
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significant difference was found for only one of twenty-seven items suggests that the two versions of the 
course had similar effects on the use of many instructional practices explored during the course. 


Table 3: Summary of Teacher Log Mean Scores by Treatment Group 



Face-to-Face (n=36) 

Online (n=49) 


Pre 

Post 

Change 

P 

Pre 

Post 

Change 

P 

Review material 

2.88 

3.12 

0.24 

0.06 

2.89 

3.05 

0.16* 

0.05 

Adapt material 

2.83 

3.08 

0.25* 

0.05 

2.89 

2.91 

0.02 

0.80 

Develop new material 

2.30 

2.55 

0.25* 

0.03 

2.21 

2.35 

0.14 

0.24 

Analyze students work from prior 
lessons 

2.75 

2.83 

0.08 

0.34 

2.70 

2.93 

0.23* 

0.03 

Examine examples of teaching in the 
form of vides, case studies, etc. 

1.31 

1.63 

0.31* 

0.01 

1.30 

1.72 

0.42* 

<0.01 

Design assessment tasks 

2.12 

2.60 

0.48* 

0.00 

2.35 

2.35 

0.01 

0.96 

Collaborate with other teachers in 
preparation for lesson 

1.86 

1.83 

-0.03 

0.84 

2.03 

2.21 

0.18 

0.17 

Solve problems and consider alternate 
strategies students may use 

2.86 

2.98 

0.12 

0.39 

2.71 

2.99 

0.28* 

0.01 

Have students do introductory drills 

2.39 

2.31 

-0.08 

0.62 

2.52 

2.30 

-0.22 

0.04 

Ask students to make conjectures 

2.70 

2.97 

0.27* 

0.03 

2.92 

3.01 

0.09 

0.39 

Ask student to use drawings or 
pictures to solve or explain a problem 

2.78 

3.14 

0.36* 

0.05 

2.88 

3.23 

0.35* 

0.01 

Ask students to use words to present 
an answer 

3.02 

3.38 

0.36* 

0.01 

3.05 

3.29 

0.24* 

0.02 

Have students review homework 

2.06 

1.82 

-0.24 

0.07 

2.03 

1.93 

-0.10 

0.40 

Work with manipulatives 

2.26 

2.81 

0.55* 

0.01 

2.75 

3.09 

0.34* 

0.01 

Ask follow-up questions to students' 
responses 

3.16 

3.46 

0.31* 

0.01 

3.13 

3.41 

0.28* 

0.00 

Lead whole class discussions 

2.91 

3.23 

0.32* 

0.01 

2.97 

2.98 

0.01 

0.91 

Demonstrate/explain concepts to 
whole class with manipulatives 

3.09 

3.14 

0.05 

0.72 

3.10 

2.95 

-0.16 

0.08 

Have students do worksheets 

2.25 

1.89 

-0.36* 

0.01 

2.09 

2.07 

-0.01 

0.91 

Have student work on independent, 
long-term projects 

1.26 

1.32 

0.06 

0.53 

1.24 

1.41 

0.18* 

0.03 

Ask students to solve problems 
presented in the context of everyday 
life 

2.51 

2.94 

0.43* 

0.01 

2.55 

3.00 

0.45* 

0.00 

Have students work independently** 

2.80 

3.01 

0.21 

0.13 

2.85 

2.41 

-0.44* 

0.00 

Have students work in pairs or small 
groups 

2.93 

3.21 

0.29* 

0.04 

2.80 

3.09 

0.29* 

0.00 

Have students respond to one another 

2.84 

2.97 

0.13 

0.39 

2.59 

2.93 

0.34* 

0.01 

Have students work together on 
problems for which there is no 
immediately obvious method or 
solution 

1.54 

2.00 

0.46* 

0.01 

1.69 

2.00 

0.31* 

0.02 

Administer a multiple-choice test/quiz 

1.18 

1.08 

-0.09 

0.19 

1.12 

1.26 

0.14* 

0.02 

Administer an open-ended test/quiz 

1.30 

1.20 

-0.09 

0.35 

1.29 

1.44 

0.15 

0.15 

Stopped class to address discipline 

1.58 

1.36 

-0.22* 

0.01 

1.60 

1.56 

-0.04 

0.52 


* Change within group statistically significant at p=.05. 
**Change between groups statistically significant at p=.05. 
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L. Student Survey Results 

Similar to the teacher logs, analyses of the student survey focused on responses to individual items. For 
each item, the mean response for students within each class was used to estimate the frequency with which a 
given instructional or learning strategy was employed within the teacher’s classroom. For each treatment 
group (face-to-face and online), the mean score for each item was then calculated across all teachers during 
each administration period (pre-course and post-course). Recall that the scale was structured such that a low 
score (1) represented frequent use of the strategy while a high score (4) indicated that the student reported 
never being exposed to the strategy. Table 4 displays the change in mean response within each group. 
Within the face-to-face group, statistically significant changes occurred for three items. Specifically, 
students reported increases in the frequency with which teachers asked them to come up with new ways to 
solve math problems and to write in words about their approach to solving a problem. Students also felt 
more strongly that their teacher tried to understand the student’s approach to doing math problems. 


For the online group, statistically significant changes occurred for five items. Students of online teachers 
reported a decrease in the frequency with which teachers had them work on math worksheets, but increases 
in the asking students to come up with new ways to solve problems, explaining how they got answers to 
problems, and working on extended math projects. Students also felt more strongly that their teacher tried 
to understand the student’s way of doing math problems. 


To examine whether the amount of change differed between the two groups, an independent samples t-test 
was performed for each item. After adjusting for multiple comparisons, no significant differences among 
groups were found. 


Table 4: Mean Score Difference between Pre- and Post-surveys by Treatment Groups 



Face-to-Face (n=35) 

Online (n=45) 


Pre 

Post 

Change 

P 

Pre 

Post 

Change 

P 

1 . My teacher asks me to come up with new 
ways to solve math problems. 

2.57 

2.76 

0.19* 

0.02 

2.66 

2.87 

0.20* 

<0.01 

2. My teacher asks me to explain how I got my 
answers to math problems. 

3.28 

3.38 

0.11 

0.08 

3.30 

3.39 

0.09* 

<0.01 

3. My teacher tries to understand my way of 
doing math problems. 

3.03 

3.19 

0.16* 

0.02 

3.13 

3.21 

0.08* 

0.03 

4. My teacher expects me to participate in class. 

3.67 

3.76 

0.10 

0.12 

3.74 

3.76 

0.02 

0.57 

5. My teacher is interested in my work even if I 
get the wrong answer. 

3.21 

3.28 

0.07 

0.25 

3.22 

3.27 

0.05 

0.09 

6. My teacher likes us to think about different 
ways to solve each math problem. 

3.11 

3.21 

0.10 

0.17 

3.14 

3.15 

0.01 

0.55 

7. My teacher asks questions in math class that 
make me think. 

3.28 

3.22 

-0.06 

0.21 

3.25 

3.27 

0.02 

0.72 

8. In math class, we work on math projects that 
last a long time. 

2.00 

1.90 

-0.09 

0.28 

2.12 

2.27 

0.15* 

0.02 

9. My teacher reviews our math homework in 
class. 

2.75 

2.76 

0.01 

0.95 

2.89 

2.89 

-0.01 

0.39 

10. My teacher helps me when s/he notices that 1 
am having trouble in math. 

3.38 

3.33 

-0.05 

0.12 

3.40 

3.44 

0.04 

0.28 

1 1 . My teacher really listens to what 1 have to 
say in math class. 

3.45 

3.46 

0.01 

0.97 

3.50 

3.53 

0.03 

0.71 

12. My teacher has us do math worksheets. 

3.16 

3.15 

0.00 

0.98 

3.31 

3.21 

-0.10* 

0.03 
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13. My teacher asks us to use drawings or 
pictures when we do a math problem. 

2.49 

2.66 

0.17 

0.08 

2.63 

2.63 

0.00 

0.55 

14. My teacher asks us to write in words our 
thoughts about our answers to a math problem. 

2.35 

2.59 

0.24* 

0.04 

2.78 

2.77 

-0.01 

0.77 

15. When 1 don't understand my math work, my 
teacher tries to help me by asking me questions. 

3.13 

3.07 

-0.06 

0.42 

3.17 

3.19 

0.02 

0.67 

16. My math teacher really makes me think. 

3.21 

3.27 

0.06 

0.26 

3.24 

3.23 

-0.01 

0.53 


* Change within group statistically significant at p=.05. 


M. Mathematics Understanding 

To examine the effect that participation in the online professional development course had on teachers’ 
knowledge of teaching mathematics, teachers completed a mathematics test that contained 15 multiple- 
choice items and 5 open-response items. Table 5 displays the mean total score for each treatment group for 
the pre- and post-course test administration. 


Within each group, total test scores increased significantly. For the face-to-face group, scores increased hy 
1.86 points (on a scale of 1-20) while scores for the online group increased 1.48 points. An independent 
samples t-test, however, indicated that the score increases did not differ significantly between the two 
groups (p = 0.48). 


Table 5: Mean scores for Pre- and Post-Tests by Treatment Group 



N 

Pre 

Face-to-Face 
Post Change 

P 

N 

Pre 

Online 

Post Change 

P 

Total Score 

36 

13.57 

15.43 1.86 

0.02 

49 

13.68 

15.16 1.48 

0.02 


VII. SUMMARY AND DISCUSSION 

The study presented here compared the effects that a face-to-face and an online version of the same course 
had on the intended outcomes of that course. For both versions, participants engaged in the same reading 
material, learning activities, writing assignments, and instructional activities in their own classrooms. The 
length of the course was also the same across both conditions (8 weeks) and the level of participation and 
interaction among participants was designed to be equivalent. The fundamental difference between the two 
versions was the medium in which participants interacted with each other and their facilitator, namely face- 
to-face or online. 


The intended learning outcomes of both courses were identical. Specifically, the course was intended to 
affect teachers’ pedagogical beliefs, their instructional practices when teaching mathematics, and their 
understanding of teaching the base ten number system. To examine the effect that this online professional 
development course had on these three outcomes, several data collection instruments were administered. A 
mathematics test was used to measure changes in teachers understanding of teaching the base ten number 
system. A survey was used to measure teachers’ pedagogical beliefs. And the combination of a teacher 
survey, student survey, and instmctional logs were used to measure teachers’ instructional practices. 


As described in greater detail above, both conditions of the course altered teachers’ beliefs about teaching 
mathematics, changed their instructional practices, and, to a lesser extent, increased their understanding of 
teaching the base ten number system. On average, there were significant increases in teachers’ confidence 
teaching mathematics, strength in their belief in student-centered instructional practices, and confidence 
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analyzing students’ problem solving strategies. As reflected in both the teacher’s instructional logs and the 
student surveys, there were also several significant changes in teachers’ instructional practices. Among 
these were asking students to develop new ways to solve math problems, trying to understand students’ 
approaches to solving math problems, asking students to describe in writing how they solved math 
problems, having students work on extended math problems, and having students work in pairs or groups. 
Finally, a significant increase occurred on a test that measured teachers’ understanding of teaching the base 
ten number system. Each of these outcomes is not surprising, particularly since teachers, on average, rated 
the quality of the course as high and reported that many components of the course provided valuable 
learning opportunities. 


While a few differences in the magnitude of change were found between the two versions, overall the 
effects of the course were similar across the two conditions. Specifically, the improvement in teacher test 
scores did not differ significantly between the face-to-face and online versions. Similarly, there were no 
statistically significant differences in the amount of change in teachers’ pedagogical beliefs. Among the 
twenty-seven instructional strategies teachers were asked to log, a significant difference was found for only 
one strategy — having students work independently. And, for the sixteen items on the student survey, there 
were no significant differences. Thus, overall the data suggests that, while both versions of the course 
affected the intended learning outcomes, the effects were similar across the two online and face-to-face 
versions of the course. This finding is consistent with previous research that generally reports similar effects 
for face-to-face and online professional development courses. 


While the effects of the course were similar across delivery modes, it is interesting to note that there was a 
significant difference in opinions about whether, after completing the course, teachers preferred to 
participate in future professional development in face-to-face environment rather than online. While 
teachers who were in the face-to-face group expressed a strong desire for the face-to-face mode, those 
teachers who were actually able to experience an online version were less enthusiastic about taking future 
courses in a face-to-face mode. Since there were no differences in teachers’ prior experiences taking online 
courses, this difference in opinion about taking future courses online or face-to-face may suggest that as 
teachers become more familiar with online professional development, they are more open to taking future 
courses online. 


Nonetheless, these findings must be placed in the context of the many limitations of this study. First, the 
study focused on only one course delivered to elementary school mathematics teachers. Had a different 
course that focused on different types of teachers or different content been employed, the outcomes may 
have been different. 


Second, both versions of the course experienced considerable attrition. On average, 36% of the teachers 
who began a course did not finish the course or the required data collection instruments. The fact that 36% 
of teachers did not complete the course may mean that the course either did not meet their needs or was too 
challenging for them. Although the characteristics of teachers who dropped out of the course did not differ 
between the two versions, it is plausible that had these teachers persisted, different findings may have 
resulted. 


Third, the course employed for this study was well designed and employed high quality reading materials 
and learning activities. In addition, the course lasted for eight weeks. While the quality of the course was not 
compared to other professional development courses, it is likely that many other courses are of shorter 
duration, employ lower quality reading material, or ask participants to engage in less effective classroom 
activities. For a course that is shorter in length or that employs materials of a lower quality, the mode of 
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delivery may have a larger impact on the outcomes of the course. 


Finally, all of the participants in this course were volunteers. Although a small stipend was offered to 
compensate participants for the time required to complete the data collection instruments, the teachers who 
participated and completed the course were likely highly motivated individuals who were sincerely 
interested in developing their mathematics teaching skills. In many cases, however, participation in 
professional development is required hy a school or district. In such cases, some teachers may he less 
motivated or engaged in the learning. If presented with an online version of the course, some of these 
teachers might he unmotivated to log into the course, complete the reading and activities, or make a minimal 
effort when completing assignments. The resulting effects of the course might then he smaller. 


Despite these limitations, this study provides further evidence that an online course designed as an 
interactive learning experience can have very positive effects on teachers knowledge, pedagogical beliefs, 
and instructional practices, and that the magnitude of these effects are similar to those that occur in a face- 
to-face environment. 


VIII. ACKNOWLEDGEMENTS 

The authors wish to thank their colleagues, Fleidi Larson, Josephine Louie and Sherry Booth, for their 
contiihutions to this work. This material is based upon work supported by the National Science Foundation 
under Grant No. 0309401 to Author Organization. Any opinions, findings, and conclusions or 
recommendations expressed in this material are those of the author and do not necessarily reflect the views 
of the National Science Foundation. 


IX. REEERENCES 

1. Carey, R., G. Kleiman, M. Russell, J. D. Venable & J. Louie. Online Courses for Math Teachers: 
Comparing Self-Paced and Facilitated Cohort Approaches. Journal of Technology, Learning, and 
Assessment 7(3): December 2008. Retrieved date from http://www.jtla.org. 

2. Russell, M., G. Kleiman, R. Carey & J. Douglas. Comparing self-pace and cohort-based online 
courses for teachers. Journal of Research on Technology in Education 41(4): 361-384, 2009. 

3. Clark, R. E. Reconsidering research on learning from media. Review of Educational Research 53: 
45-459, 1983. 

4. Clark, R. E. Media will never influence learning. Educational Technology Research and 
Development 42{2): 445-459, 1994. 

5. Bernard, R., P. Abrami, Y. Lou, E. Borokhovski, A. Wade, L. Wozney, P. Wallet, M. Eiset, & B. 
Huang. How does distance education compare with classroom instruction?: A meta-analysis of the 
empirical literature. Review of Educational Research 74(3): 379-439, 2004. 

6. Cobb, T. Cognitive efficiency: Toward a revised theory of media. Educational Technology Research 
and Development 45{4): 21-35, 1997. 

7. Kozma, R. A reply: Media and methods. Educational Technology Research and Development 42: 
11-14, 2004. 

8. Smith, P. L. & C. L. Dillon. Comparing distance learning and classroom learning: Conceptual 
considerations. American Journal of Distance Education 13: 107-124, 1999. 

9. Garrison, D. R. Cognitive presence for effective asynchronous online learning: The role of reflective 
inquiry, self-direction, and metacognition. In J. Bourne & J. Moored (Eds.), Elements of Quality 
Online Education: Practice and Direction, 47-58. Needham, MA: Sloan-C, 2003. 


Journal of Asynchronous Learning Networks, Volume 13: Issue 2 


85 


Face-to-face and Online Professional Development for Mathematics Teachers: A Comparative Study 


10. Spicer, J. Even better than face-to-face? In A. Thorson (Ed.), By your own design: A teacher’s 
professional learning guide, 32-33. Columhus, OH: Eisenhower National Clearinghouse for 
Mathematics and Science Education, 2002. 

11. Treacy, B., G. Kleiman & K. Peterson. Successful online professional development. Leading & 
Learning with Technology 30(1): 42-47, 2002. 

12. Weinberger, D. Small pieces loosely joined. Cambridge, MA: Perseus Publishing, 2002. 

13. Delfino, M. & D. Persico. Online or face-to-face? Experimenting with different techniques in teacher 
training. Journal of Computer Assisted Learning 23: 351-365, 2007. 

14. Meyer, K. A. Eace-to-face versus threaded discussions: the role of time and higher-order thinking. 
Journal of Asynchronous Learning Networks 7: 55-65, 2003. 

15. Sparks, D. Dreaming all that we might realize. In A. Thorson (Ed.), By your own design: A teacher's 
professional learning guide, 18-23. Columbus, OH: Eisenhower National Clearinghouse for 
mathematics and Science Education, 2002. 

16. Lawless, K. & J. Pellegrino. Professional development in integrating technology into teaching and 
learning: Knowns, unknowns, and ways to pursue better questions and answers. Review of 
Educational Research 77(4): 575-614, 2007. 

17. Ginsburg, A., T. Gray & D. Levin. Online professional development for mathematics teachers: A 
strategic analysis. Washington D. C.: National Center for Technology Innovation, American Institutes 
for Research, 2004. 

18. Whitehouse, P. L., L. A. Breit, E. M. McCloskey, D. J. Ketelhut & C. Dede. An overview of 

current findings from empirical research on online teacher professional development. In C. Dede 
(Ed.), Online professional development for teachers: Emerging models and methods, 13-29. 

Cambridge, MA: Harvard Education Press, 2006. 

19. Allen, M., E. Mabry, M. Mattrey, J. Bourhis, S. Titsworth & N. Burrell. Evaluating the 
effectiveness of distance learning: A comparison using meta-analysis. Journal of Communication 
54(3): 402-420, 2004. 

20. Neuhauser, C. Eeaming style and effectiveness of online and face-to-face instruction. The American 
Journal of Distance Education 16(2): 99-113, 2002. 

21. Russel, T. The no significant difference phenomenon. Montgomery, AE: International Distance 
Education Certification Center, 2001. 

22. Becker, H. Internet use by teachers: Conditions of professional use and teacher-directed student use. 
Irvine, CA: Center for Research on Information Technology and Organizations, 1999. Retrieved 
December 10, 2008, from: http://www.crito.uci.edu/tlc/findings/internet-use/startpage.htm . 

23. Russell, M., L. O’Dwyer, D. Bebell & H. Mirauda. Technical Report for the USEIT Study. 
Chestnut Hill, MA: Technology and Assessment Study Collaborative, 2004. Retrieved December 10, 
2008 from: http://www.intasc.org/PDE/useit rl l.pdf . 

X. ABOUT THE AUTHORS 

Michael Russell is an Associate Professor at Boston College’s Eynch School of Education and Director of 

the Technology and Assessment Study Collaborative. Contact information: 332 Campion Hall, Boston 

College, Chestnut Hill, MA 02467, russelmh@bc.edu . 


Rebecca Carey is a Project Director at the Education Development Center currently working for the 
Regional Education Eaboratory for the Northeast and Islands. Contact information: EDC, 55 Chapel Street, 
Newton, MA 02458, 617-618-2892, rcarev@edc.org . 


Gleuu M. Kleimau is Executive Director of the Eriday Institute for Educational Innovation and Professor of 
Educational Eeadership and Policy Studies at the North Carolina State University College of Education. He 
was Vice President and Center Director at Education Development Center, Inc. (EDC) while the research 


86 


Journal of Asynchronous Learning Networks, Volume 13: Issue 2 


Face-to-face and Online Professional Development for Mathematics Teachers: A Comparative Study 


reported in this article was conducted. Contact information: 1890 Main Campus Drive, Raleigh NC 27606, 
919-513-8509, glenn kleiman@ncsu.edu . 


Joanne Douglas Venable is a Research Associate at the Technology and Assessment Study Collahorative 
at Boston College. Contact information: 332 Campion Hall, Boston College, Chestnut Hill, MA 02467, 
jldouglas3 1 @ hotmail.com. 


Journal of Asynchronous Learning Networks, Volume 13: Issue 2 


87 


